General Procedure

• Determine a null hypothesis H_0 and an alternative hypothesis H_1
  • For two-sided test, we often have H_0: \mu = c and H_1: \mu \neq c.
  • For one-sided test,
    • we often have H_0: \mu < c and H_1: \mu \geq c.
    • or H_0: \mu > c and H_1: \mu \leq c.
• Compute the observed test statistic from data under the null hypothesis. e.g. For any normally distributed estimator \hat{theta}, we have

T = \frac{\hat{\theta} - \theta_0}{\hat{\text{se}}(\hat{\theta})}

• Given the distribution of test statistics under the null hypothesis, we compute the p-value. For a two-sided test, we have

p\text{-value} = \text{Pr}(|T| > z_{\alpha/2})

pnorm(2, mean = 0, sd = 1)

[1] 0.9772499

Type I and Type II Error

• False Positive/Type I Error:
  • we incorrectly reject the null hypothesis when the null is actually true.
  • p-value: the probability of false positive (\alpha). We usually use 0.05 as a threshold to reject the null.
• False Negative/Type II Error:
  • we incorrectly retain the null hypothesis when the null is actually false.
  • OR: we incorrectly reject the alternative hypothesis when the alternative hypothesis is true.
  • Statistical power of a study: the probability of true positive (1 - \beta).

p-value

• Interpretation of p-value: True or False
  • Does a smaller p-value mean a larger effect of X on Y?
  • Is p-value the probability that the null hypothesis is true?
  • Is p-value the probability that the alternative hypothesis is false?
  • Is p-value the probability that the randomly distributed number is at least as extreme as the observed number?

Multiple testing problem

A malicious researcher in a big platform company first test for an overall effect. Had the null been rejected, the researcher would report the finding. If the researcher fails to reject, then he run a test for a subgroup and report that result.

Assuming the two tests are independent, we have

\begin{aligned} \text{Pr}(\text{at least 1 reject}) =& 1 - \text{Pr}(\text{no reject}) \\ =& 1 - (1 - \alpha)^2 \\ =& 1 - 0.95 \times 0.95 \\ =& 0.0975 > 0.05 \end{aligned}

The Type 1 error goes up because of “hidden” multiple testing. The researcher can’t commit to the same data analysis plan under di↵erent realizations of the sample.

Power Analysis

• Key question: what is the probability of rejecting the null if the true value is equal to …? (i.e. probability of true positive)
• Potential causes of low statistical power
  • data variability
  • small sample size
• The goal of power analysis
  • After data collection: we want to exclude the possibility that it is lower power that causes our statistically (in-)significant result.
  • Before data collection: determine the sample size that reaches the minimum level of statistical power.
• General procedure
  • Determine the threshold to reject the null.
  • Compute the upper and lower bound of rejection regions for the null hypothesis.
  • Compute the probability of a test statistic greater than the upper bound or smaller than the lower bound, under the alternative true value.

In-class Exercise

• Download and load broockman_kalla_2016_w9.csv.
• Use t.test() to perform two-sample two-sided t-test for wave 0 (placebo test), wave 1, wave 2, wave3, and wave 4.
• What is the power of difference in means in wave 0? Please use power.t.test() and set the number of observation per group to 900 (n = 900), set the standard deviation to 30 (sd = 30). How can we interpret the result?
• Use geom_pointrange() in ggplot2 to plot the point estimate and confidence interval for each wave.
• Use the following formula to perform multivariate regression analysis for all waves except wave 0:

therm_trans_t* ~ treat_ind + therm_trans_t0 + vf_age + vf_racename + vf_female + vf_democrat

• Compute the confidence interval of the coefficient of treat_ind. To extract point estimate of coefficent, we need to use coef(), and to extract the standard error of coefficient, we need to use the following code:

sqrt(diag(vcov(model)))["treat_ind"]

• Plot the confidence interval for the coefficient of treat_ind in regression models. What do you find?